Method and apparatus for associating the color of an object with an event

ABSTRACT

Method and apparatus for associating the color of an object with an event preferably provides linked information for the object in any one of a plurality of succeeding video frames based only on the color value of the object. Preferably, at least one computer is coupled to a GUI and to a pointing structure, and is configured to: (i) identify coordinates of the object in a first video frame where the pointing structure has pointed; (ii) determine the color values of the identified object; (iii) store the color values of the identified object; (iv) detect an action of the object being selected in any one of the plurality of succeeding video frames; (v) determine the color values of the selected object; (vi) determine whether the selected object color values correspond to the stored color values; and (vii) automatically provide the linked information to the GUI only when the color values correspond.

This is a continuation of U.S. patent application Ser. No. 16/168,093,filed Oct. 23, 2018, which is a continuation of U.S. patent applicationSer. No. 12/472,761, filed May 27, 2009 (now abandoned), which is acontinuation of U.S. patent application Ser. No. 09/679,391, filed Oct.3, 2000 (now abandoned), the entire contents of all incorporated hereinby reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to event processing, and inparticular to associating the color of an object with an event. Portionsof the disclosure of this patent document contain material that issubject to copyright protection. The copyright owner has no objection tothe facsimile reproduction by anyone of the patent document or thepatent disclosure as it appears in the Patent and Trademark Office fileor records, but otherwise reserves all copyright rights whatsoever.

2. Background Art

When using a computer or other interactive devices a user can interactwith the device in various ways. One way is to use a mouse. The userutilizes the mouse to position a pointer somewhere on a screen. The userthen depresses a mouse button and depending on where the pointer is,something pre-programmed will happen if that area of the screen is“hot”.

In the context of the Internet, the user can position the mouse pointerover a hot spot “hyperlinked” to a web page and depress the mousebutton. This will cause the computer, set-top box, or other device tosend a signal to a server to retrieve that web page and display it onthe computer screen.

We foresee the desirability to interact in this manner in many differentcontexts, not only linking with web pages. Current schemes are limitedin the contexts that a user may interact. Before further describing thelimitations of current schemes, a discussion of event processing isinstructive.

Event Processing. “Event processing” refers to the association betweensome type of “action” and the occurrence of an “event”. Such a term isused herein in the context of a user's computing experience. The userperforms an action followed by an event, followed by a series of otheractions followed by other events.

An action occurs when a user positions a cursor in a specified locationon a computer screen, for instance using a mouse, and depressing one ofthe mouse buttons. This example shows a common type of action, but manyother actions exist as well within the context of computer eventprocessing, including the input of text, a voice command, or any othertype of action. The example action may take place in any computingenvironment, but one common environment that the action occurs is in aweb browser. A web browser is a graphical user interface that enablesusers to utilize the Internet, for instance by viewing web pages.Normally the user will position the mouse pointer over a “hyperlink” anddepress the mouse button there. This, in turn, triggers an event. Theevent in this example is to cause a server to retrieve another web page.Thus, in this example, the user interacts with the Internet by a seriesof actions followed by events.

Current event processing is disadvantageous, in part, because the typesof actions available and the events initiated by the actions arelimited. Before further discussing the drawbacks of current eventprocessing, an overview of the Internet is provided below.

Internet. The information network known as the “Internet” has becomeincreasingly popular. The Internet provides a body of information whichmay be accessed by users via computers for such purposes as business,education, and entertainment. In particular, the Internet includes alarge collection of interconnected documents, images, sounds, videos,and other forms of media which are stored in computers in a system knownas the World Wide Web or simply “the web”.

Internet resources are typically accessed in a two-way environment via anetwork connection. For example, connectivity to the network may be viaa conventional twisted-pair telephone line which has a relatively lowdata-carrying capacity (e.g., bandwidth), or via a higher bandwidthpath, which may comprise optical fiber, coaxial cable, ISDN, DSL,wireless connections, or other transmission mechanisms. To retrieve datafrom a network destination (e.g., an Internet site), a user sends amessage to the network destination to request the transfer ofinformation to the user's computer, referred to as a client. The networkdestination typically includes a computer known as a server. The serverthen sends a request signal to a source function, which may be a memorywhich is coupled to the server. The source function includes theuser-requested information which may comprise, for example, text,graphics, audio and/or video data. In response to the request signalfrom the server, the source function provides the requested informationto the client.

HTML. Source information which is stored in the source function is oftenstored in a format known as “Hypertext Markup Language (HTML)”. Thisfile or script format allows the display of text, graphics and audioinformation, and provides links to other pages of information through“hyperlinks.” Hyperlinks are strings of characters in a particularformat that specify the address of the desired page of information.

HTML is a system for marking documents to indicate how the documentshould be displayed, and how various documents should be linkedtogether. HTML is a form of Standard Generalized Markup Language (SGML),defined by the International Standards Organization. HTML specifies thegrammar and syntax of markup tags which are inserted into a data file todefine how the data will be presented when read by a computer programknown as a “web browser”. Conventional web browsers include InternetExplorer, Netscape Navigator, and others.

The data file, which is typically stored on a server, includes one ormore web pages which are visited by users who have computers which mayrun different browsers. When a page is visited, HTML data output fromthe server is downloaded to the client computer. The client computer'sbrowser processes the data to format a layout for the page so the pagemay be viewed by the user on a computer screen. Generally, HTML tagsprovide text formatting, hypertext links to other pages, and links tosound and image elements. HTML tags also define input fields forinteractive web pages.

An HTML application is made available to users on the web by storing theHTML file in a directory that is accessible to a server. Such a serveris typically a web server which conforms to a web browser-supportedprotocol known as Hypertext Transfer Protocol (HTTP). Servers thatconform to other protocols such as the File Transfer Protocol (FTP) orGOPHER may also be used, but do not support interactive HTML files.

HTTP defines a set of rules that servers and browsers follow whencommunicating with each other. Typically, the process begins when a useraccesses an icon in an HTML page which is the anchor of a hyperlink,(for instance, by positioning a cursor on the icon and depressing amouse button), or the user inputs a Uniform Resource Locator (URL) tohis or her web browser, described below. A connection is then made tothe server at the address and port number specified by the URL. Next,the browser sends a request to retrieve an object from the server, or topost data to an object on the server. The server sends a response to thebrowser including a status code and the response data. The connectionbetween the browser and server is then closed. A URL is a unique addresswhich identifies virtually all files and resources on the Internet. AURL has the form: method://server:port/path/file# anchor. The “method”of accessing the resource is the web browser-supported protocol, and mayinclude, for example, HTTP, FTP, GOPHER, TELNET, NEWS, or MAILTO. The“server:port” indicates the name of the server which is providing theresource, and is alternatively known as the Internet domain name. Forexample, many businesses will use their business name as part of theserver field. The port designation is the port number on the server, butis usually not used since a default port is assumed. The “path”indicates the directory path to the resource. The file indicates thefile name of the resource. The “anchor” indicates the named element inthe HTML document. Not all fields are required.

Consider the following URL, for example:http://www.business.com/information/profits.html# section_1. The accessmethod is HTTP, the server is www.business.com. there is no portspecified, the path is information, the file is profits.html, and theanchor is section_1. Thus, it may be seen that web browsers operate in atwo-way communication environment to access information by sending arequest signal defined by a URL command to a server, and receivinginformation in return.

Current Event Processing Schemes. In the context of the Internet, eventprocessing has been limited by the nature of the web. Originally, theweb was largely a textual body of interconnected information. Actionswere limited to depressing a mouse button on a text link which initiatedthe event of retrieving another web page from a specified URL. Thesemethods are disadvantageous because they are static. Such events have noapplication in an active environment, for instance where streaming videoor television signals are involved, and such events have no applicationoutside of a web browser (i.e., where a GUI is translating HTML commandsinto actions).

As the web evolved, other types of event processing have emerged. Inparticular, the web evolved to have a more active nature. First, thetext-based web became merged with images. The images resided in variousformats and could also be used to associate an action with an event. Theimages, however, suffered the same drawback as the text and essentiallyperformed the same function. For instance, the images were used in thesame event context (e.g., depressing a mouse button on the image) andthe same actions resulted (e.g., the image caused a server to sendanother web page to the user).

After images came more active web environments. One example came in theform of animation where a sequential series of images are displayed inthe web browser which cause the image to appear to move on the screen.To allow the user to act upon an animated image, one scheme takes eachframe of the animated image and pre-determines where on that individualframe a potential action should be associated with an event.

For instance, imagine an animation of a runner holding a baton where therunner is moving horizontally across the screen and the desired actionis to depress a mouse button when the pointer is over the baton. In thisexample, assume the animation consists of 100 images. Using one scheme,the 100 images are each individually stored in a computer and each imageis analyzed to find where the baton was on the screen.

Along with each of the 100 images is an image map specifying that inframe 1 the boundary of the baton is in a first position and in frame 2,the boundary of the baton is in a second position. To map to thesepositions, the prior art essentially draws a wire frame around the batonin each frame. Therefore, 100 wire frames must be drawn in varyinglocations. The baton in a moving image varies in position and angle.This in turn varies the shape of the wire frame in each frame of theimage.

The prior scheme is extremely disadvantageous because it requires atremendous amount of manual pre-processing to draw 100 varied size wireframes. Such pre-processing contains a large lag on overhead. Thisscheme also must store each and every potential action in a computer andthen wait for the action to occur. This technique uses a large amount ofthe computer's storage space and a large amount of time to create thewire frames. This scheme, in addition, is bound to a web browserimplementation and may only trigger events relating to the accessing ofa web page at a specified URL.

As the web has developed more recently, such schemes have become evenless advantageous. Specifically, the web has expanded away from a staticmodel and now incorporates other technologies, such as television andother signals. The web through this expansion has also moved to othermedia forms, such as “set-top boxes” and interactive television setswhere the user interacts with the web using a conventional televisionset as an output device. With these other media forms and otherenvironments, other types of GUIs (not only web browsers) and othertypes of actions (not only the retrieval of a web page from a specificURL) are desired. However, there is currently no efficient way toprocess events in such environments.

SUMMARY OF THE INVENTION

The present invention provides a method and apparatus for associatingthe color of an object with an event. In one or more embodiments, theinvention takes a streaming or static digital image frames andassociates an active, dynamic, or static object with an event by itscolor.

In one embodiment, an event is the retrieval of a document at aspecified URL and the display of the document in a web browser. Inanother embodiment, an event includes the retrieval of textual or audioinformation associated with the object. In another embodiment, the eventcomprises switching to a separate video stream in an environment usingstreaming media, such as a digital video disc (DVD) player or recorder,or other suitable video environment.

The invention defines the color in a location (termed a region ofinterest) by its digital color mathematical properties. For instance,one embodiment uses the RGB standard which separates the red, green, andblue properties of the color as numerical values. Another embodimentalso uses the HSV characteristics of the color, which define hue,saturation, and value as components of color. Another embodiment usesboth RGB and HSV characteristics. In one embodiment, the colorcharacteristics are obtained from a digital RGB frame buffer

As described above, the event links to another URL (i.e., to facilitatea transaction over the Internet) or in the case of a DVD or other videoenvironment, the event causes the video stream to transition to adifferent data track (i.e., to pull up different information) or to adifferent video stream

In one embodiment the invention uses an edge detection scheme to helpdefine the boundaries for the colors in a digital RGB frame. Thealgorithm developed gives those pixels an “edge” which is not visible tothe user, nor drawn on the screen, nor pre-processed prior to the userviewing the images formed by the pixels. The values of the pixelsdefines an edge rather than the actual drawing of a wire frame on thescreen.

Thus, one or more embodiments of the present invention create theassociations between objects and events dynamically in that the datawhich creates the association is a property of the image itself (i.e.,its color values). The present invention requires no alteration orpre-processing of the image to create the associations.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects and advantages of the presentinvention will become better understood with regard to the followingdescription, appended claims and accompanying drawings where:

FIG. 1 is a flowchart showing how to associate the color of an objectwith an event according to an embodiment of the present invention.

FIG. 2 is a flowchart showing how to define the color characteristics ofan object according to an embodiment of the present invention.

FIG. 3 is a flowchart showing how to define the color characteristics ofan object according to another embodiment of the present invention.

FIG. 4 is a flowchart showing how to define the color characteristics ofan object according to another embodiment of the present invention.

FIG. 5 is a flowchart showing how to associate the color of an objectwith an event using a table according to an embodiment of the presentinvention.

FIG. 6 is a computer execution environment where one or more embodimentsof the present invention may be implemented.

FIG. 7 shows an embodiment of the present invention where the framebuffer is utilized.

FIG. 8 shows an embodiment of the present invention where color rangesare utilized.

DETAILED DESCRIPTION OF THE INVENTION

The invention is a method and apparatus for associating the color of anobject with an event. In the following description, numerous specificdetails are set forth to provide a more thorough description ofembodiments of the invention. It is apparent, however, to one skilled inthe art, that the invention may be practiced without these specificdetails. In other instances, well known features have not been describedin detail so as not to obscure the invention.

One or more embodiments of the present invention are directed toenhancing a user's ability to interact with a computer, set-top box,interactive television set, or other processing device. According toembodiments of the present invention, the user provides input on somearea of a display device, for instance by depressing a mouse button at aspecified location. The invention then determines where the user hasdepressed the mouse button and what color value appears on the displaydevice at the time the user has depressed the mouse button. Then, theinvention initiates an event that corresponds to that color value.

Event Processing. In one or more embodiments, the invention takes astreaming or static media and associates an active, dynamic, or staticobject with an event by its given color. The event may be, for instance,the retrieval of a document at a specified URL in a web browser or theretrieval of other information in non-web browser implementations. Otherimplementations include, for instance, the viewing of media, via atelevision, DVD or other media player, set-top devices, web-TV,interactive TV, and video-games.

Take, for example, a web page implementing streaming video. In theexample, a game show host is wearing a green shirt. If a user providesinput to the screen by depressing a mouse with the cursor in a locationwhere the green shirt is located, the invention links to a URLassociated with the green shirt. Thus, the moving color may substitutefor a conventional, static, text link. In another embodiment, the gameshow host with the green shirt may be in a television show. In thisembodiment, providing input on the moving green shirt may initiate adifferent event, such as providing information about the shirt or thegame show host.

One embodiment of the present invention is shown in FIG. 1. At step 100,an action occurs. At step 110, the location of the action (a region ofinterest) is determined. Locating a region of interest is well known inthe art.

One method to locate a region of interest is to obtain an ordered paircorresponding to the location of the cursor at the moment of the action.For instance, a computer screen may be represented as a two-dimensionalgraph wherein the axis of the graph resides in the lower left handcorner of the computer screen. Thus all ordered pairs will correspond topositive values of x,y pairs on the graph.

After defining the region of interest, the value associated with a colorin the region of interest is determined at step 120. Thereafter, at step130, an event associated with that color is initiated.

Color Characteristics. In one embodiment, the invention defines thecolor by its mathematical properties. For instance, one embodiment usesthe RGB standard which separates the red, green, and blue properties ofthe color as numerical values. Another embodiment also uses the HSVcharacteristics of the color, which define hue, saturation, and value ascomponents of color. HSV characteristics further define the propertiesof a color, for instance V represents an analog of the intensity of acolor or how bright the color appears. Another embodiment of the presentinvention uses both RGB and HSV characteristics.

One embodiment defines color characteristics as shown in FIG. 2. At step200, it is determined whether an action has occurred. If no actionoccurred, step 200 repeats until an action occurs. If the action hasoccurred, it is determined where the action has occurred at step 210.

The location is shown herein with respect to a single pixel, “pixel N’.At step 220, a red component of a color for pixel A is determined. Then,at step 230, a green component of the color for pixel A is determined.Next, at step 240, a blue component of the color for pixel A isdetermined. Thereafter, at step 250, the color characteristics arecombined to form a color value for pixel A

After step 250, it is determined whether any event is associated withthe color value at step 255. If no event is associated with that colorvalue, no event is initiated and the process repeats at step 200. If anevent is associated with the color value, then at step 260 an eventassociated with the color value is initiated.

Another embodiment defines color characteristics as shown in FIG. 3. Atstep 300, it is determined whether an action has occurred. If no actionoccurred, step 300 repeats until an action occurs. If the action hasoccurred, it is determined where the action has occurred at step 310.The location is shown herein with respect to a single pixel, “pixel B”.At step 320, a hue component of a color for pixel B is determined. Then,at step 330, a saturation component of the color for pixel B isdetermined. Next, at step 340, a value component of the color for pixelB is determined.

Thereafter, at step 350, the color characteristics are combined to forma color value for pixel B. After step 350, it is determined whether anyevent is associated with the color value at step 355. If no event isassociated with that color value, no event is initiated and the processrepeats at step 300. If an event is associated with the color value,then at step 360 an event associated with the color value is initiated.

Another embodiment defines color characteristics as shown in FIG. 4. Atstep 400, it is determined whether an action has occurred. If no actionoccurred, step 400 repeats until an action occurs. If the action hasoccurred, it is determined where the action has occurred at step 410.The location is shown herein with respect to a single pixel, “pixel C”.At step 420, a hue component of a color for pixel C is determined. Then,at step 430, a saturation component of the color for pixel C isdetermined. Next, at step 440, a value component of the color for pixelC is determined. At step 450, a red component of a color for pixel C isdetermined. Then, at step 460, a green component of the color for pixelC is determined. Next, at step 470, a blue component of the color forpixel C is determined.

Thereafter, at step 480, the color characteristics are combined to forma color value for pixel B. After step 480, it is determined whether anyevent is associated with the color value at step 485. If no event isassociated with that color value, no event is initiated and the processrepeats at step 400. If an event is associated with the color value,then at step 490 an event associated with the color value is initiated.

Frame Buffer. In a computer or other comparable device, images are oftenused as a form of output. A common way that such devices handle imagesor streams of images (e.g., video) is by using a “frame buffer”. Foroffline pre-processing or for real-time processing using high speedcomputers, the RGB frame buffers reside in memory. Alternatively, aspecial high speed video board can be used to process an RGB framebuffer for real-time processing.

A video signal, for instance, comprises a series of still images shownsequentially where the series of images appears so quickly that a userperceives the series of pictures to be a continuous stream of motion. Asthe computer processes each image, data relating to the image is storedin the frame buffer. Such data includes, for instance, the color valuesassociated with each pixel at each location on the screen.

As a new image is displayed, new data about the image is stored in theframe buffer. Depending on the size of the frame buffer, many sequentialimages may be stored there before it fills up and old image data must beoverwritten.

One embodiment of the present invention takes advantage of the dataalready existing in the frame buffer, for instance to perform step 120of FIG. 1. FIG. 7 shows an embodiment of the present invention where theframe buffer is utilized.

At step 700, an action occurs. At step 710, the location of the action(a region of interest) is determined. Next, at step 720, the inventionaccess the frame buffer to obtain color values at the location of theaction when it occurred. Then, it is determined whether any event isassociated with the color value at step 730. If no event is associatedwith that color value, no event is initiated and the process repeats atstep 700. If an event is associated with the color value, then at step740 an event associated with the color value is initiated.

Events. As described above, one event is a link to another URL (i.e., tofacilitate a transaction over the Internet). Thus, one event maycomprise the retrieval of a web page from a server computer connected tothe Internet.

In the context of streaming media, an event may be the retrieval of atext page. For instance, assume the user is watching a DVD programshowing several of the great pyramids of Egypt. If the user causes anevent to occur in a region of interest defined by a specific pyramid,then the event may comprise the retrieval of a text page telling theuser exactly which pyramid this is, when it was built, the historyassociated with this particular pyramid, and any other pertinentinformation. Thus, the event would cause the DVD to move to a differentdata track (i.e., to pull up different information).

In the same great pyramid example, another event may comprise theretrieval of audio information, for instance a narrator providinginformation similar to that given in the text track in the aboveexample. Furthermore, the event may comprise the retrieval of a newmedia stream, such as the close up of the pyramid, alternate anglesshowing different views of the same pyramid, or views from inside thepyramid, for example.

In one embodiment an event comprises the retrieval of a data track. Adata track comprises information associated with the color acted upon.Assume for instance, that this embodiment of the present invention takesplace using an interactive television set. While the interactivetelevision set is operating, a game show host appears wearing a greenshirt. Then assume, for instance, that a user caused an action to occurat the location of the green shirt. In this instance, the data track maybe information about the shirt, the game show host, or any otherrelevant information.

Whether an event is associated with a color value may be determined in aplurality of ways. One such way is to use a table. An example of oneembodiment of a table is shown in Table 1.

TABLE 1 Color Value Event 100 retrieve document at URL www.info.comltest300 obtain data track at a specified location 200 initiate media streamat specified location 288 initiate audio track at specified location

The table may be arranged in a plurality of ways and may be accessed inmethods well known to those skilled in the art, including hash tables,hash functions, pointers, indices, or any other suitable form of tablelookup. In addition, different color values may initiate the same event.Alternatively, every color value on an output device may be associatedwith the same event.

FIG. 5 shows an embodiment of the present invention where a table isimplemented. At step 500, an action occurs. At step 510, the location ofthe action (a region of interest) is determined. Then at step 520, thevalue associated with a color in the region of interest is determined.Next, a table is searched to locate the color value to obtain an eventthat is paired with the color value at step 530. Then, at step 535, itis determined whether the color value is in the table. If it is not,then that color value is not linked to an event and no event isinitiated. In that case the process repeats at step 500. Otherwise, atstep 540, the event is initiated.

Color Ranges. In another embodiment of the present invention regions ofinterest (i.e., those regions where a user has performed an action) areassociated with events based on color ranges. A color range, forinstance, includes those shades of blue that correspond to the body of abluebird in a video stream. Realistic image and video streams will nothave a uniform shade of blue at all areas of the bluebird's body sincelight and other factors will cause natural variations in the perceivedcolor reflected from an object. In this embodiment, this reality isaccounted for, which gives this embodiment specific application indynamic and high-quality video streams.

Color ranges are defined by color characteristics. For instance, a colorrange may translate into values on an RGB or HSV scale, or could becombinations of both. Typically, in a computer system such values rangefrom 0-255. Thus, the bluebird's body may be defined as a region ofinterest where the color range includes a B value on the RGB scale of50-75, for instance.

Thus, whether the action occurred at an upper left or lower rightportion of the bluebird's body, the same event is initiated, despite thepotential that at these corners of the regions of interest, the colorvalues differed. All that is required in this embodiment is that bothdiffering color values fall within the defined color range.

In many images, the colors may be very similar, yet they may be aspectsof different components of the image and associated with differentevents. In one embodiment, the ranges are defined with respect to aplurality of color characteristics including RGB and HSV.

For instance, a range may be defined where the R is selected to have arange of 50-80, the G is selected to have a range of 150-160, and the Bis selected to have a range of 200-220. An example color is selected bythe user where the example color has RGB characteristics of 60, 155, and205 respectively. Another color on the screen that is associated with adifferent event may appear to be substantially identical to the examplecolor but has RGB characteristics of 190, 100, 33. Since the ranges havebeen defined by a plurality of characteristics, however, such minutedifferences in color shading are accounted for by this embodiment of thepresent invention.

In one embodiment, the characteristics of the color are defined by R, G,B, H, S, V, and R-G, and G-B, where R-G and G-B have red and green andgreen and blue components respectively. In this embodiment, any three ofthese variables is sufficient to uniquely define all colors displayed onan output device or stored in a frame buffer. To determine whether thevariables are within the color range for the purpose of determiningwhether an event will be associated with the area of interest, thefollowing algorithm is used (shown in FIG. 8).

At step 800, it is determined whether an action has occurred. If noaction occurred, step 800 repeats until an action occurs. If the actionhas occurred, it is determined where the action has occurred at step810. Then at step 820, three variables are chosen from a list ofpossible color components, for instance, R, G, B, H, S, V, R-G, or G-B.For the chosen variables, histograms are produced at step 830

A histogram is a graphical method for describing data. It is apresentation of quantitative data (i.e., color components) that havebeen summarized in a frequency distribution. The values of thequantitative variable are shown on the horizontal axis. A rectangle isdrawn above each class such that the base of the rectangle is equal tothe width of the class interval and its height is proportional to thenumber of data values in the class.

Using the histogram, the color ranges are determined at step 840, forinstance by determining the variable peak on the histogram+ or −threetimes its variance. After step 840, it is determined whether any eventis associated with the color range at step 850. If no event isassociated with that color range, no event is initiated and the processrepeats at step 800. If an event, is associated with that color range,then at step 860 an event associated with the color range is initiated.

Color Patterns. It is often beneficial to define a regions of interestthat is not uniform in color. A region of interest, for instance, may bea plaid tablecloth. In this case, it is not adequate to use a singlecolor value, nor a color range to define the region of interest. Toaccount for this, one embodiment of the present invention uses colorpatterns to define the region of interest.

A color pattern comprises a representation of the colors in the regionof interest, for instance a graphical representation, defined by a ploton an x,y axis. The varying colors in the region of interest createpeaks and valleys in the color pattern, graph representation. Bylocating patterns of peaks and valleys in such a color pattern,multi-color patterns, such as the plaid tablecloth, are used toassociate actions with events.

Edge Detection. One embodiment of the present invention uses an edgedetection scheme to define the boundaries for the colors. The colorcomponent of an object is described by its pixels. An algorithm givesthose pixels an “edge” which is not visible to the user, nor drawn onthe screen, nor pre-processed prior to the user viewing the imagesformed by the pixels. For instance, an invisible wire frame is createdto describe the shape of the given colored object to associate with theevent. But the value of the pixels defines the wire frame rather thanactually drawing a wire frame on the screen.

One embodiment of the present invention uses Quicktime, a softwarepackage available from Apple Computers™. In this embodiment, Quicktimeis used to support the use of “sprites”. A sprite is a clickable elementwithin a video stream Using Quicktime, a sprite is modified to bedefined by its color component. Thus, one or more embodiments of thepresent invention create the associations between objects and eventsdynamically, in that the data which creates the association is aproperty of the image itself (i.e., its color values). The presentinvention requires no alteration or pre-processing of the image tocreate the associations.

Embodiment of Computer Execution Environment (Hardware). An embodimentof the invention may be implemented as computer software in the form ofcomputer readable program code executed in a general purpose computingenvironment such as environment 600 illustrated in FIG. 6, or in theform of byte code class files executable within a Java RuntimeEnvironment™ running in such an environment, or in the form of bytecodes running on a processor (or devices enabled to process byte codes)existing in a distributed environment (e.g., one or more processors on anetwork). A keyboard 610 and mouse 611 are coupled to a system bus 618.The keyboard and mouse are for introducing user input to the computersystem and communicating that user input to central processing unit(CPU) 613. Other suitable input devices may be used in addition to, orin place of, the mouse 611 and keyboard 610. I/O (input/output) unit 619coupled to bi-directional system bus 618 represents such I/O elements asa printer, A/V (audio/video) I/O, etc.

Computer 601 may include a communication interface 620 coupled to bus618. Communication interface 620 provides a two-way data communicationcoupling via a network link 621 to a local network 622. For example, ifcommunication interface 620 is an integrated services digital network(ISDN) card or a modem, communication interface 620 provides a datacommunication connection to the corresponding type of telephone line,which comprises part of network link 621. If communication interface 620is a local area network (LAN) card, communication interface 620 providesa data communication connection via network link 621 to a compatibleLAN. Wireless links are also possible. In any such implementation,communication interface 620 sends and receives electrical,electromagnetic or optical signals which carry digital data streamsrepresenting various types of information.

Network link 621 typically provides data communication through one ormore networks to other data devices. For example, network link 621 mayprovide a connection through local network to local server computer 623or to data equipment operated by ISP 624. ISP 624 in turn provides datacommunication services through the world wide packet data communicationnetwork now commonly referred to as the “Internet” 625. Local network622 and Internet 625 both use electrical, electromagnetic or opticalsignals which carry digital data streams. The signals through thevarious networks and the signals on network link 621 and throughcommunication interface 620, which carry the digital data to and fromcomputer 600, are exemplary forms of carrier waves transporting theinformation.

Processor 613 may reside wholly on client computer 601 or wholly onserver 626 or processor 613 may have its computational power distributedbetween computer 601 and server 626. Server 626 symbolically isrepresented in FIG. 6 as one unit, but server 626 may also bedistributed between multiple “tiers”. In one embodiment, server 626comprises a middle and back tier where application logic executes in themiddle tier and persistent data is obtained in the back tier. In thecase where processor 613 resides wholly on server 626, the results ofthe computations performed by processor 613 are transmitted to computer601 via Internet 625, Internet Service Provider (ISP) 624, local network622 and communication interface 620. In this way, computer 601 is ableto display the results of the computation to a user in the form ofoutput.

Computer 601 includes a video memory 614, main memory 615 and massstorage 612, all coupled to bi-directional system bus 618 along withkeyboard 610, mouse 611 and processor 613. As with processor 613, invarious computing environments, main memory 615 and mass storage 612,may reside wholly on server 626 or computer 601, or they may bedistributed between the two. Examples of systems where processor 613,main memory 615, and mass storage 612 are distributed between computer601 and server 626 include the thin-client computing architecturedeveloped by Sun Microsystems, Inc., the palm pilot computing device andother personal digital assistants, Internet ready cellular phones andother Internet computing devices, and in platform independent computingenvironments, such as those which utilize the Java technologies alsodeveloped by Sun Microsystems, Inc.

The mass storage 612 may include both fixed and removable media, such asmagnetic, optical or magnetic optical storage systems or any otheravailable mass storage technology. Bus 618 may contain, for example,thirty-two address lines for addressing video memory 614 or main memory615. The system bus 618 also includes, for example, a 32-bit data busfor transferring data between and among the components; such asprocessor 613, main memory 615, video memory 614 and mass storage 612.Alternatively, multiplex data/address lines may be used instead ofseparate data and address lines.

In one. embodiment of the invention, the processor 613 is amicroprocessor manufactured by Motorola, such as the 680XO processor ora microprocessor manufactured by Intel, such as the 80X86, or Pentiumprocessor, or a SPARC microprocessor from Sun Microsystems, Inc.However, any other suitable microprocessor or microcomputer may beutilized. Main memory 615 is comprised of dynamic random access memory(DRAM). Video memory 614 is a dual-ported video random access memory.One port of the video memory 614 is coupled to video amplifier 616. Thevideo amplifier 616 is used to drive the cathode ray tube (CR 1) rastermonitor 617. Video amplifier 616 is well known in the art and may beimplemented by any suitable apparatus. This circuitry converts pixeldata stored in video memory 614 to a raster signal suitable for use bymonitor 617. Monitor 617 is a type of monitor suitable for displayinggraphic images.

Computer 601 may send messages and receive data, including program code,through the network(s), network link 621, and communication interface620. In the Internet example, remote server computer 626 might transmita requested code for an application program through Internet 625, ISP624, local network 622 and communication interface 620. The receivedcode may be executed by processor 613 as it is received, and/or storedin mass storage 612, or other non-volatile storage for later execution.In this manner, computer 600 may obtain application code in the form ofa carrier wave. Alternatively, remote server computer 626 may executeapplications using processor 613, and utilize mass storage 612, and/orvideo main memory 615. The results of the execution at server 626 arethen transmitted through Internet 625, ISP 624, local network 622 andcommunication interface 620. In this example, computer 601 performs onlyinput and output functions.

Application code may be embodied in any form of computer programproduct. A computer program product comprises a medium configured to,store or transport computer readable code, or in which computer readablecode may be embedded. Some examples of computer program products areCD-ROM disks, ROM cards, floppy disks, magnetic tapes, computer harddrives, servers on a network, and carrier waves.

The computer systems described above are for purposes of example only.An embodiment of the invention may be implemented in any type ofcomputer system or programming or processing environment.

Thus, a method and apparatus for associating the color of an object withan event is described in conjunction with one or more specificembodiments. The invention is defined by the claims and their full scopeof equivalents.

What is claimed is:
 1. A processor-implemented method for automaticallyidentifying an object in a plurality of Internet video frames andassociating the object with a link action, comprising: providing atleast one processor which executes the steps of: determining a locationin one of said Internet video frames where an action by a pointingdevice on the object has occurred on a display device; determining acolor value of the determined location on the object, including the stepof determining a color boundary of the object by providing color pixelson the boundary with an edge which is not visible to a user, the edgecomprising a portion of an invisible wire frame in which color values ofthe determined object pixels define the invisible wire frame edges, thecolor values of the invisible wire frame varying from frame to frame,the color value comprising a combination of color values for first,second, and third different colors; determining a color value range foreach of the color values for the first, second, and third differentcolors of the determined object, each color value range including acontinuous series of integer color values, the step of determining eachcolor value range including the steps of (i) determining a histogram foreach of the color values for the first, second, and third differentcolors of the determined object, (ii) determining the peak color valueof the histogram for each of the first, second, and third differentcolors, (iii) determining a variance of the peak color value of thehistogram for each of the first, second, and third different colors, and(iv) for each of the first, second, and third different colors,assigning color range values corresponding to at least one standarddeviation of the determined variance, the peak color value of eachhistogram consisting essentially of the highest number of data valuesfor each color value for each of the first, second, and third differentcolors; and automatically associating a link action with the combinedcolor value of said determined location on the object in said one videoframe, and automatically associating said link action with said combinedcolor value on the object in succeeding video frames, withoutpre-processing the invisible wire frame of said succeeding video frames.2. The method according to claim 1, wherein the at least one processorexecutes the step of determining a color value pattern for the selectedobject, including the step of locating plural alternating peaks andvalleys in the color pattern.
 3. The method according to claim 2,wherein the at least one processor executes the steps of automaticallyassociating a link decision with said color value pattern of theselected object in said one video frame, and automatically associatingsaid color value pattern peaks and valleys in succeeding video frames.4. Computer apparatus for automatically identifying an object in aplurality of Internet video frames, and associating the object with alink action, comprising: a graphical user interface (GUI); user pointingstructure; and at least one computer coupled to the graphical userinterface and to said user pointing structure, said at least onecomputer executing the steps of: determine a location in one of saidInternet video frames displayed on said GUI where said user pointingstructure has identified the object; determine a color value of theobject in said one video frame displayed on said GUI, includingdetermining a color boundary of the object by providing color pixels onthe boundary with an edge which is not visible to a user, the edgecomprising a portion of an invisible wire frame in which color values ofthe determined object pixels define the invisible wire frame edges, thecolor values of the invisible wire frame varying from frame to frame,the color value comprising a combination of color values for first,second, and third different colors; determine a color value range foreach of the color values for the first, second, and third differentcolors of the determined object, each color value range including acontinuous series of integer color values, the step of determining eachcolor value range including the steps of (i) determining a histogram foreach of the color values for the first, second, and third differentcolors of the determined object, (ii) determining the peak color valueof the histogram for each of the first, second, and third differentcolors, (iii) determining a variance of the peak color value of thehistogram for each of the first, second, and third different colors, and(iv) for each of the first, second, and third different colors,assigning color range values corresponding to at least one standarddeviation of the determined variance, the peak color value of eachhistogram consisting essentially of the highest number of data valuesfor each color value for each of the first, second, and third differentcolors; automatically associate a link action with the combined colorvalue of said determined location on the object in said one video frame;and automatically associate said link action with the combined colorvalue on the object in succeeding video frames, without pre-processingthe invisible wire frame of said succeeding video frames.
 5. Theapparatus according to claim 4, wherein the at least one computerexecutes the step of determining a color value pattern for the object,including the step of locating plural alternating peaks and valleys inthe color pattern.
 6. The apparatus according to claim 5, wherein the atleast one computer processor executes the steps of automaticallyassociating a link decision with decision with said color value patternof the object in said one video frame, and automatically associatingsaid color value pattern peaks and valleys in succeeding video frames.